Control system for controlling object using pseudo-emotions generated in the object

ABSTRACT

A control method for controlling operation of an object used by a user in an environment includes the steps of: defining pseudo-emotions of the object for deciding output of the object, in relation to the user&#39;s state; formulating emotion generation algorithms to establish the relationship between the user&#39;s state and the pseudo-emotions; formulating behavior decision algorithms to establish the relationship between input, including the pseudo-emotions, and the behavior of the object; detecting the user&#39;s state; generating a pseudo-emotion of the object based on the user&#39;s state using the emotion generation algorithms; making the object behave based on the user&#39;s state and the pseudo-emotion using the behavior decision algorithms; evaluating reaction of the user in response to the behavior of the object; and if the reaction of the user does not match the pseudo-emotion of the object in the emotion generation algorithms, adjusting at least either of the emotion generation algorithms or the behavior decision algorithms, followed by learning the adjustment. The object can detect the user&#39;s state in a visual, tactile, and auditory manner as do humans, and can act upon generation of pseudo-emotions based thereon. Thus, natural communication between the user and the object can be performed, i.e., more human like communication can be established.

BACKGROUND OF THE INVENTION

1. This invention relates to a system for controlling an objectinteracting with a user and environment, and particularly to that forcontrolling an object capable of expressing pseudo-emotions in responseto the user or environment by using algorithms, thereby creatingbehavior highly responsive to the states of the user or environment.

2. Heretofore, various controlling methods have been available forcontrolling an object in accordance with a user's demand.

3. In such controlling methods, normally, the user sets a target valueat output which the user wants, and the object is controlled in such away that the output matches the target value, while feeding the outputback to a control system which compares the feedback and the targetvalue to adjust the output. In the above, by feeding the output back tothe system to adjust the output, the output of the object to becontrolled can approach the target value, thereby achieving controlsatisfying the user's preference.

4. The aforesaid target value is set at a value basically satisfying therequest by the user who uses the object. In practice, methods of settinga target value include a method of setting an appropriate target valueby the user at every time the user uses the object, e.g., setting atemperature of an air conditioner, and a method of setting anappropriate target value by a manufacturer to satisfy a wide range ofusers when the object is manufactured, e.g., setting parameters of anengine for a vehicle when manufactured.

5. However, in conventional methods, because a principle goal is toobtain output in accordance with a target value, when the user inputs atarget value directly into an object, if an incorrect target value ismistakenly inputted, for example, the object is controlled based on theincorrect target value. Satisfactory results cannot be obtained. In theabove, no matter how accurately the air conditioner controls thetemperature, i.e., outputs the set target value, the user often mustreset the target temperature as the user notices the temperature is notthe one the user really wants after the user feels the outputtemperature controlled by the air conditioner. This is because it isdifficult for the user to precisely and numerically find the righttemperature, and to input the temperature value into the airconditioner.

6. Further, when the target value is set in advance by a manufacturer,for example, because the user who uses the object has differentcharacteristics from other users, it is impossible to set in advance auniversal target which satisfies all users.

7. As descried above, in the conventional methods, because the principlegoal is to obtain output in accordance with a target value which is setdirectly by the user or set in advance by the manufacturer, the outputis likely to be stable and predictable. However, the output may not bethe one the user wants, and may not reflect the user's intent oremotions which are not directly expressed or inputted.

8. In addition, in the conventional methods, because the principle goalis to obtain output in accordance with a target value, naturally, theoutput is likely to be predictable. Thus, if such a control system isapplied to a toy, for example, behavior of the toy is inevitablyrestricted to predictable mechanical movement. As a result, the userloses interest in or gets tired of playing with the toy.

SUMMARY OF THE INVENTION

9. An objective of the present invention is to solve the above problemsassociated with conventional control systems, and to provide a controlsystem which allows outputting an adequate value ultimately giving theuser more satisfaction than does a value obtained from the user's directorder, particularly using pseudo-emotions caused in the object inresponse to the user and environment.

10. One important aspect of the present invention attaining the aboveobjective is to provide a control method for controlling operation of anobject used by a user in an environment, said object capable ofreceiving signals of variable conditions which represent at least astate of the user and which are associated with operation of the object,said object capable of being programmed to behave in response to thereceived signals, said method comprising the steps of: definingpseudo-emotions of the object, which are elements for deciding output ofthe object, in relation to the signals; formulating emotion generationalgorithms to establish the relationship between the signals and thepseudo-emotions; formulating behavior decision algorithms to establishthe relationship between input, including the pseudo-emotions, and thebehavior of the object; detecting signals of variable conditions andinputting the signals into the object; generating a pseudo-emotion ofthe object based on the signals using the emotion generation algorithms;making the object behave based on the signals and the pseudo-emotionusing the behavior decision algorithms; evaluating reaction of the userin response to the behavior of the object; and if the reaction of theuser does not match the pseudo-emotion of the object in the emotiongeneration algorithms, adjusting at least either of the emotiongeneration algorithms or the behavior decision algorithms, followed bylearning the adjustment.

11. According to the present invention, the object is always allowed toregulate to a certain degree its behavior based on its ownpseudo-emotions. The object is formed in such a way as to adjust theemotion generation algorithms and the behavior decision algorithms andto learn them based on the reaction by the user in response to theobject's own behavior, and thus, the recognition efficiency ofrecognizing the user's intentions and/or emotions increases. Further,the object can detect the user's state in visual, tactile, or auditorymanner as do humans, and can act upon generation of pseudo-emotionsbased thereon. Thus, natural communication between humans and objectscan be performed, i.e., more human like communication can beestablished.

12. In the above, preferably, the control method further comprises thesteps of: recognizing an intention/emotional expression of the userbased on the signals of variable conditions, and using theintention/emotional expression of the user as the signals forformulating the emotion generation algorithms and for generating thepseudo-emotion of the object. The intention/emotional expression of theuser can also be used as the input for formulating the behavior decisionalgorithms and for deciding the behavior of the object. In the above,preferably, the control method further comprises the steps of deducingpreference/habit of the user from the recognized intention/emotionalexpression of the user, learning the deduced result, and using thelearned deduced result for recognizing the intention/emotionalexpression of the user.

BRIEF DESCRIPTION OF THE DRAWINGS

13.FIG. 1 is a schematic view showing a communication partner robot towhich the control system according to the present invention is adapted.

14.FIG. 2 is a schematic diagram showing the relationship between theuser and the robot.

15.FIG. 3 is a schematic diagram showing the structures of thecontroller 20 illustrated in FIG. 1.

16.FIGS. 4a, 4 b, 4 c, and 4 d show the patterns of rubbing the robot atthe four electrostatic approach-sensing sensor units illustrated in FIG.1.

17.FIG. 5 shows a map for making output values of the detection unit 23to correspond to five emotional models: neutral, happy, disgusted,angry, and sad, among the aforesaid seven emotional models.

18.FIG. 6 is a schematic diagram showing a neural network usable forgenerating the pseudo-emotions at the pseudo-emotion generation unit 32illustrated in Figure, wherein the user's state, which is recognizedbased on the user's facial expressions, gestures, rubbing/hittingbehavior, voices, or the like, is used as input, and basic emotionalmodels are outputted.

19.FIG. 7 is a schematic diagram showing an example of setting the orderof priority on each behavior based on the pseudo-emotion of the robot.

20.FIG. 8 is a schematic diagram showing facial expression patternswhich may be indicated on the display 10 to express the pseudo-emotionof the robot.

21.FIG. 9 is a schematic diagram showing a second embodiment of thepresent invention wherein the control system of the present invention isapplied to a vehicle such as a two-wheeled vehicle.

22.FIG. 10 is a schematic diagram showing the relationship between thebasic emotional models and the user's vehicle-operating state and thevehicle's driving state.

23.FIG. 11 is a schematic diagram showing a controller 220 in a thirdembodiment of the present invention, wherein the control system of thepresent invention is applied to an air conditioner.

24.FIG. 12 is a schematic diagram showing the relationship between thedetection elements, the recognized user's state, and the generatedpseudo-emotion.

25.FIG. 13 is a schematic diagram showing the processing flow ofdissatisfaction detection with reference to dissatisfaction withtemperature control.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS OF THE INVENTION

26. The control system for controlling an object using pseudo-emotionsof the present invention (hereinafter, referred to simply as “controlsystem”) will be explained with reference to several embodimentsdepicted in the Figures.

Basic Structures of Communication Partner Robot

27.FIG. 1 is a schematic view showing a communication partner robot(hereinafter, referred to simply as “robot”) to which the control systemaccording to the present invention is adapted, and FIG. 2 is a schematicdiagram showing the relationship between the user and the robot.

28. This robot 1 generates its own pseudo-emotions based on the user'sconditions, user's intention or emotions, and the like, and expressesthe pseudo-emotions by several behavioral means. Based on a responsefrom the user in response to the expression of its own pseudo-emotions,the robot determines whether the generated pseudo-emotions are correct,and also determines whether the user understands and recognizescorrectly the behavior created by the pseudo-emotions. In this way, therobot is controlled to undergo learning adequate pseudo-emotions to begenerated and adequate behavior for expressing the pseudo-emotions andto evolve. By doing the above, the robot can develop its ownpseudo-emotions and expression of the pseudo-emotions throughcommunications with the user.

Sensing Means

29. The robot 1 comprises a CCD camera 2 as a visual detection means, apressure-sensing sensor 4 and an approach-sensing sensor 6 as touchdetection means, and a microphone 8 as a hearing-detection means. Byusing these sensing means 2, 4, 6, and 8, the robot 1 detects by sensethe state of the user, such as a tone of voice, facial expressions, andgestures, and the operational environments where the robot is used.

30. The CCD camera 2 is installed on the top of the head and can be setin any direction via a universal joint. For example, the robot can becontrolled in such a way that the robot automatically moves toward anobject, such as a human and animal, which is a cause or source ofinformation such as changes in temperature and sound. Image informationsuch as facial expressions of the user and surrounding environments issupplied to a controller 20.

31. The pressure-sensing sensor 4 may be installed in the lower front ofthe robot 1 so that when the robot has actual contact with an obstacle,such information is provided to the controller 20.

32. In addition, the approach-sensing sensor 6 is comprised of fourunits located on the top of the robot's body (i.e., the back of therobot 1 if the robot is considered to have a face, a head, a body, legsor driving means). The four units are electrostatic approach-sensingsensor units 6FL, 6FR, 6RL, and 6RR disposed in the front left, frontright, rear left, and rear right, respectively. These sensors provide tothe controller 20 information regarding the state of ON or OFF and theduration of each ON and OFF.

33. The microphone 8 is installed on the side of the head of the robot1, and provides information to the controller 20 upon collectingsound/voices arose around the robot 1.

Behavioral Means

34. The robot 1 comprises a display 10 to display its own facialexpression and other specific information, a speaker 12 to output words,notes, roars, or effective sounds, and a drive means 14 to move inautonomic manner. By using the above, the robot 1 can act in accordancewith the user's intent as necessary, while expressing its ownpseudo-emotions.

Controller

35. As described above, the robot 1 generates its own pseudo-emotions,recognizes the user's intent and desires (emotional expression), andrecognizes an operable range, by using the built-in controller 20, basedon information regarding the user's state and/or surroundingenvironments obtained by each of the sensing means 2, 4, 6, and 8. Basedon the generated pseudo-emotions, actions of the display 10, the speaker12, and the drive means 14 are desired, and the robot 1 starts an actionto achieve the target in accordance with the user's intent or startsemotional behavior giving priority to expressed emotions.

36. In addition, the robot 1 is controlled by the controller 20 in sucha way as to evaluate adequacy of the generated pseudo-emotions andadequacy of actions by detecting by sense the user's reaction caused byits own behavior, and to learn the evaluation results, therebyundergoing evolution.

Structures of Controller

37. The structures of the controller 20 will be explained further below.FIG. 3 is a schematic diagram showing the controller 20.

38. The controller 20 receives information from the aforesaid sensuousdetection means 2, 4, 6, and 8, and information from an externalinformation source 9 such as a network as necessary.

39. The controller 20 comprises a facial expression detection unit 21, agesture detection unit 22, a rubbing/hitting detection unit 23, asound/voice detection unit 24, and a surrounding environment detectionunit 25, and by using these detection units 21-25, the controller 20detects information about the state of the user and/or the surroundingenvironment.

40. In practice, the facial expression detection unit 21 and the gesturedetection unit 22 visually detect the user's expression and/or gesture,using an appropriate image recognition system.

41. The rubbing/hitting detection unit 23 tactilely detects the user'sactions of rubbing and/or hitting the robot 1, based on inputinformation from the pressure-sensing sensor 4 and the approach-sensingsensor 6.

42. Further, the rubbing/hitting detection unit 23 conducts two-stepdetection: detection of contacted location and detection of beingrubbed, i.e., the unit 23 detects first whether the user rubs the robot1, and if the user rubs the robot, second how the user rubs the robot 1.The detection of contacted location is conducted in such a way thatamong the electrostatic approach-sensing sensors, if either sensor unit6FL (front left) or 6FR (front right) is ON, the contact is judged to belocated in the front; if either sensor unit 6RL (rear left) or 6RR (rearright) is ON, the contact is judged to be located in the rear; if eithersensor unit 6FL (front left) or 6RL (rear left) is ON, the contact isjudged to be located on the left; if either sensor unit 6FR (frontright) or 6RR (rear right) is ON, the contact is judged to be located onthe right, thereby detecting which part of the approach-sensing sensor 6is touched by the user. Rubbing detection is established when thecontacted location moves from the front to the rear, from the right tothe left, or the like. FIGS. 4a, 4 b, 4 c, and 4 d show the patterns ofrubbing the robot 1 at the four electrostatic approach-sensing sensorunits. In the above, in rubbing detection, a manner of rubbing by theuser is detected based on the duration of activation of each sensorunit: how long each sensor unit is kept ON or OFF. For example, when theuser does not move the hand touching the sensor unit(s) for a long time,it is judged that the robot 1 is being held (FIG. 4c); when the userrubs right and left at appropriate intervals without releasing the hand,it is judged that the robot 1 is being rubbed (FIG. 4a); when the userrubs in one direction as if rubbing along fur and releases the hand whenreturning, it is judged that the robot 1 is being rubbed (FIG. 4d); whenany sensor unit is activated (an ON signal is detected) once for a veryshort time while all of the sensor units are OFF, it is judged that therobot is suddenly being hit by the user; when ON signals are detectedcontinuously at a certain interval, it is judged that the robot 1 isbeing hit repeatedly. Further, in addition to the approaching-sensingsensor 6, by installing a pressure-sensing sensor at the same locationas the approach-sensing sensor 6, it can be detected whether the usertouches the robot 1 softly or roughly, or how strong the user hits therobot 1 if hitting the robot 1.

43. The sound/voice detection unit 24 auditorily detects the user'svoice and/or sounds around the robot, based on information from themicrophone 8, and analyzes or recognizes voice information, using anappropriate voice recognition means: e.g., recognition of the contentsof a conversation by the user, identification of the individual user, oridentification of sound around the robot.

44. Further, the surrounding environment detection unit 25 tactuallyrecognizes information about obstacles around the robot based on inputinformation from the pressure-sensing sensor 4, and visually recognizesinformation about the obstacle by an appropriate image recognition meansbased on input image information from the CCD camera 2.

Recognition Units

45. The controller 20 generates pseudo-emotions at a pseudo-emotiongeneration unit 32 upon recognition of the user's intent and emotionalexpression at an intent/emotion recognition unit 31 based on thedetection results by the aforesaid detection units 21-25. The controller20 also recognizes an actually movable range of the robot 1 at a movablerange recognition unit 33.

Intent/Emotion Recognition Unit

46. In practice, the aforesaid intent/emotion recognition unit 31 maycomprise a neural network, for example, which receives information aboutthe user's preferences and habits obtained from a preference/habitlearning unit 35 described later, as well as information about the stateof the user obtained from the facial expression detection unit 21, thegesture detection unit 22, and the sound/voice detection unit 23, andwhich outputs the user's intention/emotional expression based on theinput information, thereby recognizing information about the user'sintention/emotional expression based on the output value. Further inpractice, when the aforesaid intention/emotion recognition unit 33 iscomprised of a neural network(s), the neural network can be constructedin such a way as to allow selective recognition of the user's emotionalexpression by causing the neural network to undergo learning in advance,for example, the relationship between several patterns of emotions andinformation including the user's preference/habits and the state of theuser. The several patterns of emotions are obtained by categorizing inadvance the emotional expressions of the user into several patterns ofemotions such as neutral, disgusted, happy, sad, surprised, angry, andfearful. The information on the state of the user is obtained from thefacial expression detection unit 21, the gesture detection unit 22, andthe sound/voice detection unit 23, and the information on the user'spreferences and habits is obtained from the preference/habit learningunit 35.

47. Regarding recognition of the user's intentions using a neuralnetwork, selective recognition of the user's intentions can beestablished by causing the neural network to undergo advance learningthe relationship between pieces of information including the user'sstate and preferences/habits, and the contents of intentions. Thecontents of intentions, which are learned in advance by the neuralnetwork, are those the user possibly requests to the robot 1 such as“stop”, “run”, and “return”.

48. The information on the user's intentions and/or emotionalexpressions recognized at the intention/emotional expression recognitionunit 31 is used as one piece of information for generatingpseudo-emotions and as information for evaluating adequacy of thegenerated pseudo-emotions at the pseudo-emotion generation unit 32, andis used as information for creating target-achievement motion inaccordance with the user's intentions/emotional expressions and asinformation related to reaction from the user in response to the robot'sbehavior at a behavior decision means 40, and is further used as teacherdata related to the user's preferences/habits at the preference/habitlearning unit 35.

49. The intention/emotional expression recognition unit 31 may beconstituted to learn and adjust recognition algorithms in such a way asto eliminate a discrepancy between the estimated response from the userin response to the robot's own behavior which is originally a responseto the state of the user, and the user's actually evaluated intentionsand/or emotional expressions.

50. Further, preferably, the patterns of emotional expressions and thecontents of the user's intentions may be recognized by subdividing thepatterns (e.g., disgusted and happy) and the contents (e.g., “run” and“stop”) into several levels. In the above, when recognized bysubdividing the user's intentions and/or emotional expressions,adjustment of the recognition algorithms can be achieved in thesubdivided levels of each of the patterns and the contents ofintentions. In practice, for example, when the user orders the robot 1to “run”, the robot 1 recognizes the user's intention “run” in level 2(e.g., run normally) based on various states such as the facialexpression of the user when saying “run”, thereby behaving accordingly.As a result, if the robot 1 recognizes that the user's response showsdissatisfaction, the robot judges that there is a discrepancy betweenthe user's recognized intention and the user's actual intention, and therobot changes the outcome of the recognition of the user's intention tothe level where the user is judged to be satisfied (e.g., level 3: runfast, or level 1: run slowly), thereby learning the result. Byconstructing the structure as above, the robot 1 is able to implementthe user's order, e.g., “run”, by further recognizing the contents whichthe user does not expressly order, e.g., the speed or a manner ofrunning.

51. In the above, the recognition at the intention/emotional expressionrecognition unit 31 need not be constructed by a neural network, but canbe constructed by other means such as a map which can make variousstates of the user detected by the detection units correspond to thecategorized emotional patterns.

Pseudo-Emotion Generation Unit

52. The pseudo-emotion generation unit 32 stores basic emotional modelsobtained by categorizing pseudo-emotions required of the robot 1 intoseveral emotional patterns which are, in this embodiment, neutral,disgusted, happy, sad, surprised, angry, and fearful. The pseudo-emotiongeneration unit 32 selectively generates a pseudo-emotion from the aboveseven basic emotional models in accordance with the current situation,by using a map or a function which connects information on the user'sstate, information on the user's intentions/emotional expressions, andthe aforesaid each emotional model, or by using a neural network whichhas learned the relationship between information on the user's state,information on the user's intention/emotional expression, and theaforesaid each emotional model.

53. For example, FIG. 5 shows a map for making output values of thedetection unit 23 to correspond to five emotional models: neutral,happy, disgusted, angry, and sad, among the aforesaid seven emotionalmodels. In the map, values obtained from the detection unit 23 aredistributed in the vertical axis “pleasant-unpleasant” and in thehorizontal axis “sleeping-busy” under the conditions that if rubbedcontinuously, it means “pleasant”, if held in place, it means“unpleasant”, if touched often, it means “busy”, and if left alone, itmeans “sleeping”. According to this map, for example, if the usercontinuously rubs the robot 1, the robot 1 displays happiness as itspseudo-emotion, if the user holds the robot 1 in place a few times, therobot 1 displays disgust as its pseudo-emotion, if the user holds therobot 1 in place many times, the robot 1 displays anger as itspseudo-emotion, and accordingly, the pseudo-emotions are generated whichcan correspond to expressions or behavior of dislike and expressions orbehavior of joy generally found in animals including humans.

54. In the above, in addition to the corresponding relationship betweenthe output information of the rubbing/hitting detection unit 23 and theemotional models, the information on the state of the user or theinformation on the user's intention/emotional expression from the otherdetection units 21, 22, and 24 can be used in combination with therelationship. For example, even if the manner of rubbing is in thedisgusted range, if the user's state is judged to be good based on thefacial expression detection unit 21 and/or the sound/voice detectionunit 24, it is judged that the user is simply playing with the robot andholding the robot, the ultimate pseudo-emotion of the robot remainsneutral, not disgusted. Accordingly, the map, which makes theinformation on the user's state and the information on the user'sintention/emotional expression correspond to the seven basic emotionalmodels, can be constructed.

55. In the above, if generation of the pseudo-emotions at thepseudo-emotion generation unit 32 is conducted using a neural network,as shown in FIG. 6, the user's state (such as laughing or angry)recognized based on the user's facial expressions, gestures,rubbing/hitting behavior, voices, or the like is used as input, andbasic emotional models are outputted (six models in FIG. 6). The neuralnetwork is made to learn in advance the relationship between such inputand output, and to output, as parameters, coupling loads wn (n=1−6) ofeach basic emotional model corresponding to the user's state. Based onthe coupling load of each basic emotional model, an ultimatepseudo-emotion is determined using, for example, a multi-dimensionalmap.

56. As described above, if the pseudo-emotion generation unit 32 isformed by a control logic capable of learning, the pseudo-emotiongeneration unit 32 may be constructed so as to recognize the user'sreaction in response to robot's own behavior based on the informationobtained from the intention/emotional expression detection unit 31; toevaluate adequacy of the pseudo-emotion generated based on the user'sreaction; and if the generated pseudo-emotion is judged to be unnatural,to adjust the relationship between input information and outputinformation in such a way as to generate a natural pseudo-emotion inaccordance with the user's state and/or the user's intention/emotionalexpression at the moment; and to thereby learn the results ofadjustment.

57. Preferably, this pseudo-emotion generation unit 32 can generate thepseudo-emotions by subdividing each of the aforesaid seven basicemotional models into several levels. For example, the happy model canbe subdivided into several levels to generate pseudo-emotions containingthe degree of happiness (e.g., a degree from “very happy” to “slightlyhappy”). In this case, the pseudo-emotion generation unit 32 may beformed so as to adjust pseudo-emotion generation algorithms persubdivided level and to undergo learning accordingly.

58. The information on the generated pseudo-emotions is outputted to thebehavior decision means 40, and used, at the behavior decision means 40,as information for performing emotional behaviors, as standardinformation for putting priority to several behaviors as describedlater, and as comparison information for evaluating the informationrelated to the user's reaction obtained from the intention/emotionalexpression recognition unit 31.

Ambient Environment Memory Unit and Movable Range Recognition Unit

59. An ambient environment memory unit 34 successively memorizesinformation related to the ambient environment inputted from the ambientenvironment detection unit 25. A movable range recognition unit 33recognizes the range where the robot 1 can actually move based on theinformation from the ambient environment detection unit 25 and theambient environment memory unit 34.

60. The information related to the recognized movable range is outputtedto the behavior decision unit 40.

Preference/Habit Learning Unit

61. The preference/habit learning unit 35 constantly receivesinformation related to the user's intentions/emotional expressionsrecognized at the intention/emotional expression recognition unit 31,determines the user's preferences and/or habits based on the user'sintentions and/or emotional expressions and undergoes learning the same,and outputs the information to the intention/emotional expressionrecognition unit 31 as one piece of information for recognizing theuser's intentions/emotional expressions.

62. In addition, the output from the preference/habit learning unit 35is also inputted into an information-searching unit 36.

Information-Searching Unit, Information Memory Unit, InformationIntegration/Processing Unit

63. The information-searching unit 36 searches for adequate informationfor the user's preference/habits obtained at the preference/habitlearning unit 35 using the external information source 9, and makes theinformation memory unit 37 memorize the result. An informationintegration/processing unit 38 integrates the searched information andthe memorized information, processes the information (e.g., selectsnecessary information), and outputs the result to the behavior decisionmeans 40.

Behavior Decision Means

64. As described above, all of the information related to the user'sintention/emotional expression recognized at, for example, eachrecognition unit and each generation unit, the information related tothe pseudo-emotions of the robot, the information related to the movablerange, and the information related to the user's intentions/emotionalexpressions are used as standards for target-achievement action, theinformation related to the pseudo-emotions of the robot is used asstandards for emotional behaviors, and the information related to themovable range is used as standards for obstacle-avoiding action.

65. The behavior decision means 40 comprises a behavioral typeartificial intelligence system, and is formed by an appropriate growthsystem which evaluates a discrepancy between the pseudo-emotionsactually arising in the robot 1 and the pseudo-emotions of the robot 1recognized by the user, and makes the behavioral type artificialintelligence system undergo adjustment, learning, and/or evolving.

66. In practice, the behavior decision means 40 decides which actions,i.e., target-achievement action, emotional behavior, orobstacle-avoiding action, or wandering when no order is given, the robot1 is made to activate, by setting the order of priority on eachbehavior.

67. In practice, the target-achievement action includes an actiondirectly in accordance with the target the user intends, i.e., an actionsuch that when the user orders “run”, the robot “runs”.

68. The emotional behavior includes actions basically residing inemotions, such as an expression of happiness by indicating a smilingface on the display 10 while displaying dancing behavior by going backand forth or turning around using the drive means 14 when thepseudo-emotion is “happy”, and an expression of anger by indicating aangry face on the display 10 while rushing straight using the drivemeans 14 when the pseudo-emotion is “angry”.

69. Further, the object-avoiding action includes an action to avoid anobstacle, and the wandering when no order is given includes a repeatingaction such as going forward or changing directions without targets.

70. Setting the order of priority on each behavior (i.e., emotionalbehavior, target-achievement action, obstacle-avoiding action, andwandering) is conducted based on the pseudo-emotion of the robot 1generated at the pseudo-emotion generation unit 32.

71.FIG. 7 is a schematic diagram showing an example of setting the orderof priority on each behavior based on the pseudo-emotion of the robot.As shown in this figure, in this embodiment, when the pseudo-emotion is“neutral”, the obstacle-avoiding action takes priority over the otheractions, followed by the target-achievement action, i.e., the order ofpriority is set in order to make the robot act as an obedient robot,which suppresses robot's own pseudo-emotional drive, toward the user.When the pseudo-emotion is “disgusted”, “happy”, or “sad”, theobstacle-avoiding action takes priority over the other actions, followedby the emotional behavior, i.e., the order of priority is set in orderto allow the robot to express its own pseudo-emotions by, e.g., goingaway angry or being sad due to the robot's action which was adverse tothe user's intention, or treading on air due to happiness. Further, whenthe pseudo-emotion is “angry”, “surprised”, or “fearful”, the emotionalbehavior takes priority over the other actions, i.e., the order ofpriority is set in order to allow the robot to express its ownpseudo-emotions such as an surprised expression when the robot keepsgoing forward even if the robot hits an obstacle.

72. After completion of selection of behavior based on the above orderof priority, the behavior decision means 40 operates the display 10, thespeaker 12, and the drive means 14 in order to take action suitable forthe contents of behavior selected by the robot.

73. In practice, the display 10 selects a facial expression suitable forthe pseudo-emotion of the robot at the moment among several facialexpression patterns such as those indicated in FIG. 8, and displays theselected facial expression. In addition, when the user requests thedisplay of certain information, the display 10 can display theinformation by request instead of the facial expressions, or both theinformation and the facial expressions.

74. The speaker 12 outputs a voice suitable for the pseudoemotion of therobot at the moment (such as laughing sounds when the pseudo-emotion is“happy”), an answer in response to the user's request when thetarget-achievement action is given priority, or adequate effectivesounds, by synthesizing appropriate sounds.

75. The drive means 14 drives according to the behavior taking priority.

76. In the above, a decision method will be explained in detail when thebehavior decision means 40 selects emotional behavior. The behaviordecision means 40 learns in advance the relationship between each of thebasic seven emotional models (preferably each level of each model) andplural patterns of behavior corresponding thereto. For example, when thepseudo-emotion of the robot becomes “angry” as a result of a situationwhere the user fixedly touches the approach-sensing sensor 6 andmaintains the situation, the display 10 is made to display the facialexpression of “angry” and the drive means 14 is made to go back toescape from the user's hand and to stop when the hand is detached fromthe sensor. When the pseudo-emotion is “happy”, the display 10 is madeto display the facial expression of “smiling face” and the drive means14 is made to move in combination of going back and forth and turningaround.

Learning at Behavior Decision Means

77. The behavior decision means 40 judges how the user recognizes thepseudo-emotion of the robot 1, which was aroused in response to theuser's reaction which arose in response to robot's own behavior, basedon the information obtained from the intention/emotional expressionrecognition unit 31, and the behavior decision means 40 evaluates thebehavior decision algorithms at the behavior decision unit 40, based onthe discrepancy between the pseudo-emotion recognized by the user andthe pseudo-emotion actually aroused in the robot.

78. If no discrepancy is found, the behavior decision unit 40 is judgedto optimally act expressing the pseudo-emotions, and the behaviordecision algorithms remain the same. If the actual pseudo-emotion of therobot and the pseudo-emotion recognized by the user are different, thediscrepancy is determined, and the relationship between thepseudo-emotion and the behavior in the behavior decision algorithms atthe behavior decision means 40 is adjusted to eliminate the discrepancy.For example, despite the pseudo-emotion of anger aroused in the robot 1,the user recognizes that the pseudo-emotion is “disgusted”.

79. In the pseudo-emotions, for example, each of the basic sevenemotional models is subdivided into levels 1 to 3, and the behaviordecision algorithms have learned the relationship between the emotionalmodel “joyful” and the behavior as follows:

80. (1) “Joyful level 1 (slightly joyful)=going around with a smilingface.”

81. (2) “Joyful level 2 (joyful)=dancing around while laughing with asmiling face.”

82. (3) “Joyful level 3 (very joyful)=dancing around while loudlylaughing with a smiling face.”

83. If the robot recognizes based on the user's reaction “the userrecognizes that the robot is very joyful (the pseudo-emotion: Joyfullevel 3)” as a result of behavior expressing “joyful level 2” by therobot, the behavior decision means 40 learns the relationship betweenthe emotional model “joyful” and the behavior in order to match thepseudo-emotion of the robot being recognized by the user and thepseudo-emotion aroused in the robot as follows:

84. (1′) “Joyful level 1 (slightly joyful)=running around with a smilingface.”

85. (2′) “Joyful level 2 (joyful)=dancing around with a smiling face.”

86. (3′) “Joyful level 3 (very joyful)=dancing around while laughingwith a smiling face.”

87. As described above, by adjusting the relationship between thepseudo-emotion and the behavior in the behavior decision algorithmsbased on the reaction by the user, appropriate emotional behaviorcorresponding to the pseudo-emotions can be established.

Effects Exhibited in The Embodiment

88. As explained above, because the robot 1 sets the order of priorityon several behavior patterns based on the pseudo-emotions of the robot,the robot 1 is always allowed to regulate to certain degree its behaviorbased on its pseudo-emotions, not only when the emotional behavior takespriority over the other behaviors.

89. The robot 1 described above is formed in such a way as to adjust theuser's intention/recognition algorithms, emotion generation algorithms,and behavior decision algorithms and to learn them based on the reactionby the user in response to the robot's own behavior, and thus, therecognition efficiency of recognizing intentions/emotions increases,thereby establishing pseudo-emotions and emotional expressions.

90. The robot 1 described above detects the user's state in visual,tactile, and auditory manner as do humans, and acts upon generation ofpseudo-emotions based thereon. Thus, natural communication betweenhumans and robots can be performed, i.e., more human like communicationcan be established.

91. Further, the robot 1 sets the order of priority on behaviors basedon its pseudo-emotions. Thus, the robot sometimes acts entirelyunexpectedly, and will not make the user tired of playing with therobot.

Second Embodiment Vehicle

92.FIG. 9 is a schematic diagram showing a second embodiment of thepresent invention wherein the control system of the present invention isapplied to a vehicle such as a two-wheeled vehicle.

93. In this embodiment, the user's state is detected based on thedriving state detected by, for example, a throttle sensor, a brakesensor, or a sensor sensing handle bar operation (hereinafter, referredto as “handle bar sensor”).

94. In practice, based on detection information from the throttlesensor, information related to the user's throttle operation isdetected; based on detection information from the brake sensor,information related to the user's brake operation is detected; and basedon detection information from the handle bar sensor, information relatedto the user's handle bar operation is detected.

95. An intention/emotional expression recognition unit 131 may comprisea neural network, for example, which receives information about theuser's preference and habits obtained from a preference/habit learningunit 134, as well as information related to throttle operation, brakeoperation, and handle bar operation (hereinafter, these pieces ofinformation are referred to as “information related to avehicle-operating state”), and which recognizes the user'sintentions/emotional expressions based on the user's vehicle-operatingstate and the user's preference/habits. The basic principle of theintention/emotional expression unit 131 is the same as in theintention/emotional expression unit 31 in the first embodiment, andthus, its detailed explanation will be omitted.

96. A pseudo-emotion generation unit 132 stores basic emotional modelsobtained by categorizing pseudo-emotions required by the vehicle intoseveral emotional patterns which are, for example, neutral, disgusted,happy, sad, surprised, angry, and fearful. The pseudo-emotion generationunit 132 receives information related to the user's vehicle-operatingstate as well as information related to vehicle's state itself such asvelocity, engine r.p.m's., engine temperature, and the remaining fuelvolume, and generates its own pseudo-emotion using a neural network or amap which has learned the relationship between the input information andthe aforesaid basic emotional models.

97. The relationship between the basic emotional models and the user'svehicle-operating state and the vehicle's driving state itself isdetermined by using as standards, for example, the changes in velocity,the throttle-operating state, the throttle angle, the accelerationstate, the brake-operating state, and the changes in the handle barangle and by making them correspond to the basic emotional models, asshown in FIG. 10. In the relationship indicated in FIG. 10, it is judgedthat the higher the indices of the speed change, the throttle-operatingstate, the throttle angle, the accelerating state, and thebrake-operating state, the rougher the driving operation by the userbecomes, whereby the basic emotional model becomes “angry”. Also, it isjudged that the higher the indices of the brake-operating state, thehandle bar-operating state, and the speed change, the higher the degreeof the user's tiredness becomes, whereby the basic emotional modelbecomes “sad”.

Preference/Habit Learning Unit and Other Units

98. The basic principles of a preference/habit learning unit, aninformation-searching unit, an information memory unit, and aninformation integration/processing unit are the same as in the firstembodiment, and thus, detailed explanation will be omitted.

Behavior Decision Means

99. A behavior decision means 140 receives information related to theuser's intentions/emotions obtained at the intention/emotionalexpression unit 131 as standards for target-achievement action, andreceives information related to the pseudo-emotions of the vehicle asstandards for emotional behavior, and gives priority to thetarget-achievement action and the emotional behavior based on theaforesaid pseudo-emotion, thereby determining either thetarget-achievement action or the emotional behavior, based on the orderof priority.

100. In the above, for example, the target-achievement action includesan action which simply targets the user's throttle operation, brakeoperation, or handle bar operation, thereby operating the throttle orthe like.

101. Emotional behaviors include actions urging as priority the user totake a rest using signals or voices when the pseudo-emotion is “sad”.

102. Practical operation means include driving operation subjected tocontrol related to driving performance of the vehicle such as controlover ignition timing, fuel flow, thermostat, ABS, TCS, and suspension;an action of warning the user in a visual or auditory manner, e.g.,warning when driving at excessive speeds or when running out of gas; andan action pf releasing the user's stress by providing trafficinformation, weather reports, or news flashes.

103. In addition, the operation means also include a means forindicating the pseudo-emotion of the vehicle, wherein the user can judgethe vehicle's pseudo-emotions and can operate the vehicle in such a wayas to make the vehicle happy, thereby performing adequate driving.

104. The intention/emotional expression unit 131, the pseudo-emotiongeneration unit 132, and the behavior decision means 140 adjust, as doesthe controller in the first embodiment, the user's intention/recognitionalgorithms, emotion generation algorithms, and behavior decisionalgorithms and learn the adjustments, using the user's reaction inresponse to its behavior as evaluation standards, and thus, therecognition efficiency of recognizing intentions/emotions increases,thereby establishing its pseudo-emotions and emotional expressions.

Third Embodiment Air Conditioner

105.FIG. 11 is a schematic diagram showing a controller 220 in a thirdembodiment of the present invention, wherein the control system of thepresent invention is applied to an air conditioner.

106. In this embodiment, the controller 220 detects the user's state andsurrounding environment, and based on the detected information,generates the pseudo-emotion of the air conditioner itself, andrecognizes the user's intentions/emotional expressions.

107. The air conditioner comprises, for example, a remote controller forswitch operation, a microphone, a CCD camera, and a pyroelectric sensoras means for detecting information related to a state of the user(s),and a clock, an indoor environment sensor (detecting, for example,temperature, humidity, and degree of cleanness of air), and an outdoorenvironment sensor (detecting, for example, temperature, humidity,barometric pressure, wind velocity, sunshine, rainfalls, and snowfalls)as means for detecting using environment, and further, an externalinformation source such as a network.

108. The controller 220 detects intentions of user(s) based oninformation obtained form the external information source, the operationswitch remote controller, and the microphone; detects voices of theuser(s) using an appropriate voice recognition means based oninformation obtained from the microphone; detects sounds made inside andoutside the room including noises; detects facial expressions andcomplexion of the user(s) based on information obtained from the CCDcamera; detects movement of user(s), existence of user(s), and thenumber of user(s) based on information obtained from the CCD camera andpyroelectric sensor; detects indoor environment based on informationobtained from the indoor environment sensor; and detects outdoorenvironment or weather based on information obtained from the outdoorenvironment sensor.

109. An intention/emotional expression recognition unit 231 and apseudo-emotion generation unit 232 recognize intentions/emotionalexpressions of the user(s) and generate pseudo-emotions of the airconditioner itself, based on the detected state of the user(s) and thedetected using environment by using the aforesaid detection means.

110. In practice, as shown in FIG. 12, if the pseudo-emotion generationunit 232 for generating pseudo-emotions detects that the room is quietbased on information from the microphone, detects no movement by user(s)based on information from the proelectric sensor, and detects that theroom is dark based on information from the CCD camera, and after theabove conditions continue for a given time period, if the proelectricsensor detects movement of user(s), the CCD camera detects light, themicrophone detects the sound of a key in a door, sounds of a TV orradio, the sound of a curtain closing, and/or sound of a playback of avoice mail device, then the pseudo-emotion generation unit recognizesthe arrival of user(s) at home based on a combination in part of theforegoing detections. Further, the pseudo-emotion generation unitrecognizes the number of user(s) who arrived using an appropriate imagerecognition means based on information from the CCD camera, identifiesthe individual user(s) using an appropriate voice recognition meansbased on information from the microphone, recognizes whether themovement is active or slow, recognizes the indoor environment such asroom temperature, humidity, sunniness, and tranquilness based oninformation from the indoor environment sensor, and detects the outdoorenvironment such as temperature and weather based on information fromthe outdoor environment sensor, and based on the foregoing recognizedinformation, if, for example, the indoor and outdoor temperatures arejudged to be low at arrival of the user(s), then the pseudo-emotiongeneration unit generates a pseudo-emotion “want to warm up”, and if theindoor and outdoor temperature is judged to be low and outdoor winds arejudged to be strong at arrival of the user(s), then the pseudo-emotiongeneration unit generates a pseudo-emotion “want to warm up quickly”.

111. The above-described pseudo-emotion generation can be achieved bystoring pseudo-emotion models required of the air conditioner, and bycausing a neural network to learn the relationship between thepseudo-emotions and the various recognized data or by using a mapdefining correspondence between the pseudo-emotions and the recognizedinformation.

112. In the above, the relationship between the pseudo-emotions and therecognized information further includes, for example, “want to cool downquickly” if it is hot when user(s) arrive(s), “want to providerefreshing and dehumidified air” if it is hot and humid when user(s)arrive(s), “want to prevent getting hot” if there are many people orusers, and “want to prevent getting chilled or too cold” if it is quiet.

113. Like the pseudo-emotion generation unit 232, an intention/emotionalexpression recognition unit 231 may comprise a neural network whichlearns in advance the relationship between the recognized informationand the intentions/emotional expressions of the user(s) to output to abehavior decision means 240 the intentions/emotional expressions of theuser(s) at the moment such as actual desire of ventilation which is notexpressly ordered by the user(s) but which is recognizable based oninformation such as the user's complexion related to feeling of theuser, in addition to the particularly set temperature controlled via theswitch operation remote controller.

114. In the above, the information on the intentions and emotionalexpressions of the user(s) recognized at the intention/emotionalexpression recognition unit 231 is also used as teacher data for apreference/habit learning unit 235, and the preference/habit learningunit 235 undergoes learning the preferences and/or habits of the user(s)by adding them, and outputs the result of learning to theintention/emotional expression recognition unit 231 as one piece of theinformation used for recognizing the intentions/emotional expressions ofthe user(s).

115. The behavior decision means 240 receives information related to thepseudo-emotions of the air conditioner itself in addition to informationrelated to the recognized intention/emotional expressions of theuser(s); conducts control such as cooling/heating capacity control, windflow control, wind direction control, air purifier control, andventilation control based on a control value which is obtained bysumming a target-achievement operation value based on theintentions/emotional expressions of the user(s) and a pseudo-emotionoperation value based on the pseudo-emotions; and as necessary, informsthe user(s) of temperature and/or humidity by using a display or voiceoutput.

Detection of Dissatisfaction and Learning

116. In addition, the intention/emotional expression recognition unit231 also recognizes the intention/emotional expression of the user(s)regarding dissatisfaction based on reaction by the user(s) in responseto the behavior or performance of the air conditioner regulated by theaforesaid behavior decision means 240; and based on the recognitionresult, adjusts the recognition algorithms at the intention/emotionalexpression recognition unit 231 and the behavior decision algorithms atthe behavior decision means 240, followed by learning the changes.

117. The dissatisfaction detection will be explained with reference todissatisfaction with temperature control. As shown in FIG. 13, actionsexpressed when the user is dissatisfied with the temperature, such asre-setting the remote controller, pronouncing a word “hot” or “cold”,and performing gestures such as fanning himself or herself andshrinking, are detected by detection means such as a operation switch,microphone, and CCD sensor; and according to the detected information,the operation value is recognized or the meaning of words is recognizedby using an appropriate voice recognition means. Further, behavior orcomplexion of the user is detected using an appropriate imagerecognition means, and if the user is judged to perform an actionobserved when dissatisfied with temperature, the user's dissatisfactionis established.

118. The controller adjusts the aforesaid recognition algorithms and thebehavior decision algorithms based on the detected dissatisfaction, andlearns the results, and also learns the user's preferences/habits,whereby the recognition efficiency of recognizing intentions/emotionsincreases, and its pseudo-emotions and emotional expressions arebecoming established.

Effects Exhibited in The Third Embodiment

119. As explained above, because the air conditioner recognizes theintentions/emotional expression of the user(s) by taking thepreference/habits of the user(s) into account, temperature control, forexample, with consideration of each individual user'spreferences/habits, can be conducted for a user who has a habit toalways raising the set temperature from the temperature the useractually desires, or a user who cannot tolerate the heat and prefers arelatively low temperature. As a result, the air conditioner allowsproviding satisfactory environment to the user(s) whose unexpressedintentions/emotional expressions are satisfied.

120. Further, according to the above third embodiment, because the airconditioner is constructed so as to conduct, for example, temperaturecontrol by adding the emotional behavior based on its ownpseudo-emotions to the target-achievement action based on the user'sintentions/emotional expressions, the air conditioner can provideenvironment optimal to the state of the user(s) even if no order isgiven.

Other Embodiments

121. In the first embodiment, the CCD camera, for example, is used asmeans for detecting the state of the user(s) or the using environment.However, the detection means can be any means which are capable ofdetecting the state of the user(s) or the using environment.

122. The structure of the robot in the first embodiment is not limitedto the one indicated in FIG. 1. For example, the robot can beincorporated into a stuffed animal or a doll. Further, the use of therobot described in the first embodiment is not limited, and the robotcan have various uses, e.g., a toy or a device for medical use.

123. Further, the drive means in the first embodiment is not limited,and for example, can be embodied as hands and feet or even a tail.

124. In the second embodiment, the driving state of the user is detectedusing the throttle sensor, brake sensor, and handle bar sensor, andbased on the detected driving state, the intentions/emotionalexpressions are recognized and the pseudo-emotions are generated.However, the detection means are not limited thereto, and for example,indication showing a state of the user himself or herself such as theuser's heartbeat or perspiration or facial expressions are detected, andbased on the results, the intentions/emotional expressions can berecognized and the pseudo-emotions can be generated. Further,recognition of the intentions/emotional expressions and generation ofthe pseudo-emotions can be performed by using a combination of the stateof driving operation and the state of the user himself or herself.

125. In addition, the detection means in the third embodiment are notlimited, and any means which are capable of detecting the state of theuser or outdoor environment can be used.

126. The object to be controlled in the present invention is notlimited, and any objects, as in the first, second, and thirdembodiments, such as an outboard engine in a vessel, a robot used inmachine tools, a motor used in electrically-driven vehicles, or the likecan be controlled by adopting the control system of the presentinvention based on the same principle as in the aforesaid embodiments.

127. It will be understood by those of skill in the art that numerousvariations and modifications can be made without departing from thespirit of the present invention. Therefore, it should be clearlyunderstood that the forms of the present invention are illustrative onlyand are not intended to limit the scope of the present invention.

What is claimed is:
 1. A control method for controlling operation of anobject used by a user in an environment, said object capable ofreceiving signals of variable conditions which represent at least astate of the user and which are associated with operation of the object,said object capable of being programmed to behave in response to thereceived signals, said method comprising the steps of: definingpseudo-emotions of the object, which are elements for deciding output ofthe object, in relation to the signals; formulating emotion generationalgorithms to establish the relationship between the signals and thepseudo-emotions; formulating behavior decision algorithms to establishthe relationship between input, including the pseudo-emotions, and thebehavior of the object; detecting signals of variable conditions andinputting the signals into the object; generating a pseudo-emotion ofthe object based on the signals using the emotion generation algorithms;making the object behave based on the signals and the pseudo-emotionusing the behavior decision algorithms; evaluating reaction of the userin response to the behavior of the object; and if the reaction of theuser does not match the pseudo-emotion of the object in the emotiongeneration algorithms, adjusting at least either of the emotiongeneration algorithms or the behavior decision algorithms, followed bylearning the adjustment.
 2. The control method according to claim 1 ,further comprising the steps of: recognizing an intention/emotionalexpression of the user based on the signals of variable conditions, andusing the intention/emotional expression of the user as the signals forformulating the emotion generation algorithms and for generating thepseudo-motion of the object.
 3. The control method according to claim 1, further comprising the steps of: recognizing an intention/emotionalexpression of the user based on the signals of variable conditions, andusing the intention/emotional expression of the user as the input forformulating the behavior decision algorithms and for deciding thebehavior of the object.
 4. The control method according to claim 2 ,further comprising the steps of: recognizing an intention/emotionalexpression of the user based on the signals of variable conditions, andusing the intention/emotional expression of the user as the input forformulating the behavior decision algorithms and for deciding thebehavior of the object.
 5. The control method according to claim 2 ,further comprising the steps of deducing preference/habit of the userfrom the recognized intention/emotional expression of the user, learningthe deduced result, and using the learned deduced result for recognizingthe intention/emotional expression of the user.
 6. The control methodaccording to claim 3 , further comprising the steps of deducingpreference/habit of the user from the recognized intention/emotionalexpression of the user, learning the deduced result, and using thelearned deduced result for recognizing the intention/emotionalexpression of the user.
 7. The control method according to claim 3 ,wherein the behavior decision algorithms output at least a decision oftarget-achievement action corresponding to the recognizedintention/emotional expression of the user and a decision of emotionalbehavior corresponding to the pseudo-emotion of the object, wherein theorder of priority is set for each decision based on the pseudo-emotionof the object, and the behavior decision algorithms output eitherdecision based on the order of priority.
 8. The control method accordingto claim 1 , wherein the signals of variable conditions are detectedusing sense-based detecting means.
 9. The control method according toclaim 1 , wherein the signals of variable conditions are selected fromthe group consisting of the user's facial expression, gesture, manner oftouching, and voice condition.
 10. The control method according to claim1 , wherein the emotion generation algorithms are formulated using basicemotional models corresponding to plural emotional patterns, and outputat least one emotional model as the pseudo-emotion.
 11. The controlmethod according to claim 10 , wherein the emotion generation algorithmsare formulated to select one emotional model from the basic emotionalmodels based on the signals of variable conditions, and to output theemotional model as the pseudo-emotion.